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FOREWORD 


Computer-based  instruction  (CBI)  is  being  touted  as  a  significant  instruc¬ 
tional  technology  for  the  United  States  Armed  Services.  CBI's  potential  as  a 
training  medium  is  dependent  on  instructional  design  and  implementation  that 
meets  the  needs  of  the  end  user — the  military  instructor.  Training  developers 
without  previous  CBI  experience  may  have  trouble  developing  a  useful  CBI  prod¬ 
uct,  however,  because  they  do  not  know  the  pitfalls.  This  research  report 
discusses  the  issues  and  lessons  learned  by  the  Fort  Knox  Training  Technology 
Field  Activity  (TTFA)  in  fielding  CBI  training  programs  at  the  U.S.  Army  Armor 
School,  Fort  Knox,  KY.  Twenty-five  recommendations  on  managing,  developing, 
implementing,  and  evaluating  a  CBI  project  are  made  in  the  report. 

The  work  described  in  this  research  report  was  part  of  the  Fort  Knox  Field 
Unit's  research  program  to  apply  new  training  technology  to  armor  skills  train¬ 
ing.  A  Memorandum  of  Agreement  covering  the  application  of  training  technology 
to  armor  skill  training  was  signed  by  the  U.S.  Army  Armor  School,  the  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC),  and  the  U.S.  Army  Research  Institute 
(ARI)  on  4  November  1983.  This  agreement  was  renewed  on  28  March  1987.  The 
Technical  Director  of  the  U.S.  Army  Armor  School  has  been  briefed  on  the  recom¬ 
mendations  cited  in  this  report,  and  these  recommendations  are  being  applied  to 
additijnal  CBI  projects  being  developed  and  implemented  at  Fort  Knox. 
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ISSUES  IN  DEVELOPING  AND  IMPLEMENTING  COMPUTER-BASED  INSTRUCTION  FOR  MILITARY 
TRAINING 
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EXECUTIVE  SUMMARY 


Requirement : 

Computer-based  instruction  (CBI)  is  being  touted  as  a  significant  instruc¬ 
tional  technology  for  the  United  States  Armed  Forces.  CBI's  potential  as  a 
training  medium  is  dependent  on  quality  instructional  design  and  planning. 
However,  training  developers  without  previous  CBI  experience  may  have  trouble 
developing  a  quality  CBI  product  because  they  do  not  know  the  potential  prob¬ 
lems  they  could  face. 


Procedure : 

Based  on  a  series  of  ongoing  CBI  projects  managed  by  the  Training  Tech¬ 
nology  Field  Activity  (TTFA)  at  Fort  Knox,  KY,  this  report  discusses  issues  in 
managing,  developing,  and  implementing  CBI  programs.  This  discussion  focuses 
on  problems  that  the  TTFA  had  with  management,  development,  implementation, 
and  evaluation  issues.  Recommendations  for  future  CBI  training  projects  are 
made . 


Findings : 

Based  on  the  lessons  learned  from  TTFA's  experiences,  25  recommendations 
are  made.  These  recommendations  stress  careful  planning  at  the  different  de¬ 
velopmental  stages,  coordinating  the  work  of  many  people,  having  all  partici¬ 
pants  co-locate  at  the  instructional  site,  following  the  SAT  (Systems  Approach 
to  Training)  method  of  courseware  development,  meeting  the  instructors'  needs, 
and  conducting  careful  evaluations.  Actively  involving  school  management  per¬ 
sonnel  in  the  CBI  project  is  also  stressed. 


Utilization  of  Findings: 

The  recommendations  made  in  this  report  are  currently  being  applied  and 
refined  at  Fort  Knox  in  the  development  and  implementation  of  new  CBI  programs 
to  support  the  training  of  Armor  noncommissioned  and  commissioned  officers. 
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ISSUES  IN  DEVELOPING  AND  IMPLEMENTING 
COMPUTER-BASED  INSTRUCTION  FOR  MILITARY  TRAINING 

INTRODUCTION 


Computer-based  instruction  (CBI)  is  being  touted  as  a  significant  in¬ 
structional  technology  for  the  United  States'  Armed  Forces.  Orlansky  (1985) 
has  noted  that  CBI  is  particularly  applicable  to  training  military  personnel 
in  a  significant  number  of  the  over  10,000  armed  services  skill  training 
courses  offered  in  such  areas  as  electronics,  maintenance,  and  communica¬ 
tions.  Many  reports  on  CBI  (e.g..  Gray,  1986;  Montague,  Wulfeck,  &  Ellis, 
1983;  Shlechter,  in  press)  have  noted  that  CBI's  potential  as  a  training 
medium  is  dependent  upon  quality  instructional  design  and  quality  implementa¬ 
tion. 

Training  developers  without  previous  experience  with  CBI  training  pro¬ 
grams  may  have  trouble  developing  a  quality  CBI  product  because  they  do  not 
know  the  potential  pitfalls  facing  them.  Very  few  reports  exist  which  would 
help  them  by  documenting  specific  problems  with  fielding  this  training  medium 
in  a  military  setting.  This  report  then  discusses  the  lessons  learned  by  the 
Training  Technology  Field  Activity  (TTFA)  at  Fort  Knox,  KY  in  fielding  CBI 
programs. 

TTFA  is  a  cooperative  effort  of  the  US  Army  Training  and  Doctrine  Command 
(TRADOC),  the  US  Army  Armor  School  (USAARMS),  and  the  US  Army  Research  Insti¬ 
tute  (ARI) .  The  TTFA  mission  is  to  systematically  identify,  introduce,  and 
evaluate  technology  to  improve  the  efficiency  and  effectiveness  of  Army 
Training.  TTFA  developed  CBI  training  for  the  MOS  19K  BNCOC  program  (Basic 
Non-Commissioned  Officer  Course  for  Ml  Tank  Commanders).  Courseware  was 
written  for  the  following  lessons;  Land  Navigation,  Diagnostic  Test  and  Reme¬ 
dial  Training,  Military  Communications,  Call  for  and  Adjust  Indirect  Fire, 

NBC  Defense,  Mine  Warfare,  and  Tank  Fire  Commands.  Voice  recognition  sys¬ 
tems,  videodisc  systems,  and  a  MicroTICCIT  System  II  (Microcomputer  Time- 
Shared  Interactive  Computer-Controlled  Television)  were  used  as  the  training 
media.  These  programs  were  TTFA's  initial  efforts  with  CBI  training. 

TTFA  personnel  have  found  that  fielding  a  CBI  training  program  involves 
dealing  with  issues  found  in  the  development  of  any  other  training  program. 
These  issues  were  managing,  developing,  implementing,  and  evaluating  the 
instructional  program.  However,  TTFA  has  found  that  these  Issues  are  more 
complex  for  fielding  a  CBI  training  program  than  for  most  other  instructional 
systems.  For  instance,  the  courseware  must  fit  the  unique  constraints  asso¬ 
ciated  with  a  particular  hardware  system,  software  system,  and  authoring 
language.  The  complexities  and  approaches  involved  in  dealing  with  these 
issues  in  the  TTFA  context  will  be  explored  in  this  report. 

It  must  be  noted  that  some  of  the  TTFA's  approaches  and  difficulties 
which  were  experienced  might  not  generalize  to  other  CBI  implementations. 
First,  TTFA  was  not  an  operational  activity  of  the  Armor  School.  It  was  an 
ad  hoc  group  pulled  together  for  the  specific  TTFA  mission.  As  such,  it  was 
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outside  the  Armor  School  and  subject  to  all  the  problems  of  an  outside  agency 
trying  to  develop  and  implement  training  in  the  School.  Second,  very  few  CBI 
projects  will  involve  as  many  different  programs  and  instructional  devices  as 
did  the  TTFA  project.  Third,  these  CBI  projects  had  to  deal  with  some  unique 
constraints.  For  instance,  because  of  limited  in-house  personnel  resources, 
TTFA  had  to  rely  on  contract  personnel  for  training  design  and  development. 


MANAGING  THE  CBI  PROJECT 

The  previously  mentioned  complexities  inherent  in  TTFA's  CBI  projects 
required  that  the  utmost  care  be  taken  in  managing  these  CBI  projects.  Man¬ 
aging  these  training  projects  included  coordinating  the  efforts  of  many  peo¬ 
ple  from  different  organizations  and  monitoring  contractual  efforts, 
courseware  production,  and  the  instructional  delivery  site. 


TTFA's  Team  Approach 


These  CBI  projects  required  a  sizeable  team  effort  involving  ARI,  TRADOC, 
and  Armor  School  TTFA  representatives,  BNCOC  instructors,  subject  matter 
experts  (SMEs)  from  the  Armor  School,  contractual  courseware  developers,  and 
the  hardware  developer. 

Each  of  these  groups  had  a  different  role  in  the  courseware's  development 
and  implementation.  ARI  representatives  monitored  the  different  contractual 
efforts  and  conducted  the  preliminary  trials  of  the  courseware.  TRADOC  rep¬ 
resentatives  monitored  the  implementation  of  the  courseware  in  the  BNCOC 
classroom  and  conducted  the  in-course  validation  checks  (see  the  evaluation 
section  for  a  description  of  these  evaluational  approaches).  The  instructors 
provided  information  on  specific  course  content.  The  SMEs  provided  answers 
on  correct  military  doctrine  and  reviewed  the  interim  courseware  products  to 
insure  that  proper  military  doctrine  was  used.  Contractual  courseware  devel¬ 
opers  then  designed  and  developed  the  courseware  for  these  tasks.  All  the 
courseware,  except  the  Land  Navigation  and  Diagnostic/Remediation  lessons, 
were  developed  by  one  firm.  Project  managers  (TTFA)  made  decisions  about 
developmental  and  implementation  issues  based  upon  advice  from  members  of 
various  steering  committees.  Steering  committees  included  personnel  from 
ARI,  TRADOC,  and  the  Armor  School. 


Problems  with  TTFA's  Team  Approach 


Unfortunately,  there  were  some  problems  with  this  team  approach.  Because 
they  were  located  at  different  sites,  these  professionals  seldom  directly 
interacted  with  each  other.  The  prime  courseware  developer,  for  instance, 
directly  interacted  with  the  BNCOC  instructors  on  only  two  occasions.  Infor¬ 
mation  about  the  instructors'  desires  were  provided  to  this  firm  by  the  per¬ 
sonnel  monitoring  the  contract.  Such  triangular  communication  frequently 
occurred  during  the  course  of  the  supporting  contract,  a  situation  which 
resulted  in  frequent  miscommunicat ions  among  the  participants. 
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Correspondingly,  the  different  participants  did  not  understand  the  con¬ 
straints  faced  by  the  other  participants.  The  prime  courseware  developer 
never  realized  the  SMEs'  time  constraints  nor  did  the  SMEs  understand  this 
courseware  developer's  time  constraints.  Needing  a  quick  review  on  an  in¬ 
terim  product,  the  courseware  developer  requested  that  these  products  be 
examined  immediately.  However,  the  SMEs,  having  other  military  duties,  needed 
more  time  to  properly  review  these  products.  The  courseware  developer  thus 
experienced  unexpected  delays  in  completing  the  final  product. 

TTFA  also  experienced  some  difficulties  in  achieving  a  consensus  among 
the  team  members.  The  SMEs  and  the  BNCOC  instructors  frequently  provided 
different  training  content  information.  For  instance,  the  BNCOC  instructors 
and  the  SMEs  provided  quite  different  information  for  the  "coding  numbers' 
task  in  the  Military  Communications  lesson.  After  considerable  discussion 
with  the  different  team  members,  the  TTFA  decided  to  accept  the  SME's  views. 
Again,  the  time  table  for  developing  and  implementing  this  courseware  was 
delayed. 

Rapid  turnover  among  the  SMEs  also  caused  problems.  A  new  SME  often  had 
different  notions  about  the  subject  matter  from  those  expressed  by  his  prede¬ 
cessors.  TTFA  received,  for  instance,  several  different  opinions  on  complet¬ 
ing  an  NBC-1  report  from  different  SMEs. 

Some  other  problems  were  found  in  these  projects  due  to  the  team  approach 
employed.  First,  the  military  instructors  tended  to  regard  these  projects  as 
imposed  solutions  because  decisions  about  their  course  were  not  being  made  by 
them.  An  early  effort  to  forestall  this  problem  by  co-locating  TTFA  with  the 
BNCOC  instructors  was  defeated  when  TTFA  was  removed  from  the  BNCOC  site. 
Secondly,  those  individuals  who  monitored  the  courseware  development  were  not 
the  people  who  were  responsible  for  implementing  these  projects.  The  Armor 
School,  for  instance,  had  to  change  some  of  the  completed  courseware  to  meet 
specific  implementation  needs. 

Recommendations :  Some  of  these  problems  can  be  eliminated  in  future  CBI 

ojects  by:  a)  having  more  direct  meetings  between  participants  and  insist¬ 
ing  that  all  agencies  attend  all  meetings;  b)  surfacing  all  constraints  to 
development  and  implementation;  c)  having  a  flexible  time  table  for  complet¬ 
ing  the  project;  d)  involving  instructional  personnel  more  directly  in  the 
courseware  development  process;  e)  having  a  trained  permanent  support  staff 
to  help  develop  the  courseware;  and  f)  having  the  same  personnel  who  are 
responsible  for  implementing  the  products  monitor  the  different  contractual 
efforts.  Other  recommendations  about  methods  for  (and  problems  with)  coordi¬ 
nating  the  team  effort  needed  for  fielding  a  CBI  project  will  be  discussed 
throughout  this  report. 


The  Roles  of  the  Project  Manager 


TTFA  has  found  that  the  project  manager's  role  is  vital  in  developing  and 
Implementing  a  CBI  training  package.  This  person  must  coordinate  the  team's 
efforts,  devise  a  realistic  time  table,  and  understand  the  different  issues 
involved  in  the  project. 


3 


ARI's  representative  was  the  manager  for  developing  the  projects  while 
the  TRADOC  representative  was  the  manager  for  implementing  the  training  pro¬ 
grams.  These  people  and  their  staffs  were  quite  knowledgeable  about  military 
training  procedures;  however,  they  had  little  expertise  in  computer  systems 
and  courseware  production  methods.  More  knowledge  of  CBI  systems  and  the 
courseware  production  methods  would  have  helped  them  to  better  understand  the 
constraints  faced  by  the  instructional  delivery  system  operator,  courseware 
contractor,  and  the  hardware  developer.  With  more  expertise  they  would  have 
realized  that  the  voice  recognition  system  could  only  be  used  for  very  lim¬ 
ited  purposes,  e.g.,  simple  wcrds  and  sentences.  Time  and  money  were  spent 
trying  to  make  the  voice  recognition  system  understand  complex  military  com¬ 
munications  sequences,  when  it  was  really  only  adequate  for  training  fire 
commands . 

Also,  one  person  should  have  been  the  project  manager.  Sometimes  these 
two  project  managers  had  trouble  coordinating  their  activities.  The  imple¬ 
mentation  manager,  because  of  other  commitments,  did  not  participate  in  some 
developmental  decisions.  These  two  groups  also  rarely  reviewed  the  course¬ 
ware  together  and  had  different  issues  to  examine  when  reviewing  the  course¬ 
ware.  For  example,  the  implementation  team  was  more  interested  in  the  length 
of  the  "Call  for  Fire"  program  than  was  the  developmental  team.  The  imple¬ 
mentation  manager  should  have  been  responsible  for  the  entire  project  as  this 
person  was  responsible  to  the  Armor  School  for  fielding  this  project. 

Recommendation :  Managers  need  some  knowledge  of  CBI  systems  and  course¬ 
ware  production  methods.  A  single  project  manager  is  better  than  several 
sharing  responsibility. 


The  Terminal  Site  Manager 

TTFA  also  discovered  the  importance  of  the  delivery  system  site  manager. 
This  person  had  the  following  functions  to  perform:  l)  installing  the  devel¬ 
oped  courseware  into  the  system;  2)  making  sure  that  the  system  had  enough 
disc  space  for  the  different  courseware;  3)  making  copies  of  the  different 
courseware;  4)  changing  the  courseware  after  the  contract  had  expired;  5) 
scheduling  use  of  the  system;  6)  troubleshooting  problems  with  the  systems, 
and  7)  training  other  people  (e.g.,  proctors  and  helpers)  to  use  the  system. 
Most  of  these  functions  were  quite  involved.  For  example,  changing  the 
courseware  required  expert  knowledge  of  a  special  authoring  language.  These 
changes  also  had  to  be  made  rapidly  to  meet  implementation  deadlines.  Sched¬ 
uling  the  use  of  the  system  required  coordinating  regular  "shut-down"  periods 
with  the  facility's  maintenance  personnel  and  coordinating  activities  with 
the  evaluators  and  BNCOC  instructors.  Such  personnel  could  not  use  the  sys¬ 
tem  at  will. 

The  TTFA  project  managers  did  not  initially  appreciate  the  importance  of 
Lhe  site  manager.  The  site  manager  was  given  a  three-week  block  of  instruc¬ 
tion  on  the  basics  of  using  the  MicroTICCIT  (Microcomputer  Time-Shared  Inter¬ 
active  Computer-Controlled  Information  Television)  System  II.  This 
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instructional  program  was  found  to  be  insufficient  for  teaching  the  special 
authoring  language  to  the  level  of  proficiency  required. 

Recommendations :  The  site  manager  must  have  previous  CBI  experience. 
Site  managers  should  also  be  thoroughly  trained  in  operating  and  programming 
the  CBI  system  before  the  project  is  started. 


The  Terminal  Site 


TTFA  also  found  out  the  importance  of  securing  an  appropriate  site  for 
the  computer  terminals.  They  had  problems  with  acquiring  a  terminal  site 
which  did  not  need  many  alterations.  The  space  provided  was  not  already 
configured  for  the  delivery  system.  The  chosen  terminal  site  thus  had  to  be 
re-conf igured  and  extensive  architectural  changes  had  to  be  made  in  the  new 
facility.  The  site's  ventilation  had  to  be  redone  in  order  to  accommodate 
the  computer  hardware.  Some  additional  furnishings  (e.g.,  dividers  between 
terminals)  were  also  needed.  Additional  expenditures  were  incurred  by  TRADOC 
to  make  this  room  suitable  for  TTFA's  use. 

Another  problem  with  preparing  this  terminal  site  for  CBI  use  was  that 
the  TTFA  could  not  find  any  written  guidance  governing  such  alterations.  TTFA 
did  not  know  the  items  to  consider  and  the  people  to  contact  in  making  these 
alterations.  The  TTFA  team  started  with  the  post  engineers,  since  these 
alterations  included  additional  outlets,  electrical  buffered  power,  and  addi¬ 
tional  air  conditioning.  The  post  engineers  led  the  team  to  the  following 
people:  the  Provost  Marshal  for  site  security,  the  Director  of  Automated 
Information  Management  for  information  security,  and  the  Fire  Marshal  for 
removal  of  the  automatic  sprinkler  system  and  installation  of  a  Halon  system. 

This  new  site  was  also  problematic  because  it  was  not  located  in  or  next 
to  the  BNCOC  "school-house".  The  evaluation  team  for  the  Military  Communica¬ 
tions  courseware,  for  example,  noticed  that  approximately  fifteen  minutes 
were  wasted  at  the  beginning  of  each  instructional  period  waiting  for  stu¬ 
dents  to  find  the  room  and  then  to  settle  down.  This  placement  of  the  termi¬ 
nals  also  caused  the  instructors  problems  when  they  forgot  to  bring  some 
instructional  aids,  e.g.,  an  Army  Field  Manual.  They  either  had  to  leave  the 
classroom  to  get  the  materials  or  do  without.  Hence,  a  poorly  located  termi¬ 
nal  site  can  cause  instructional  problems. 

Recommendations :  Great  care  should  be  given  to  selecting  an  appropriate 
CBI  terminal  site.  This  site  should  need  few  alterations  to  accommodate  CBI 
training.  This  site  also  must  be:  1)  located  in  or  near  the  "school-house”; 
2)  kept  at  a  temperature  suitable  for  operating  the  terminals;  and  3)  secured 
easily.  The  issues  involved  in  selecting  and  preparing  the  site  must  be 
resolved  before  the  hardware  is  purchased. 


Co-Location  of  All  Parties  to  a  CBI  Project 

Because  the  courseware  development  contracts  did  not  specify  on-site 
production,  the  courseware  contractors  were  located  away  from  the  SMEs,  BNCOC 
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instructional  personnel,  BNCOC  students  and  the  TTFA  staff.  Serious  problems 
resulted  from  the  courseware  contractors*  absence  from  the  instructional 
site. 

One  problem  was  the  previously  discussed  triangular  communications  among 
the  different  participants.  Correspondingly,  problematic  communications 
arose  when  the  SMEs  and  the  courseware  developers  had  to  communicate  directly 
over  the  phone.  Few  people,  for  instance,  have  the  skill  to  articulate  over 
a  phone  the  procedures  involved  in  a  tank  commander's  call  for  artillery 
fire. 

Another  problem  with  the  courseware  developers'  location  was  that  the 
courseware  project  manager  could  not  directly  supervise  the  final  products. 
Despite  daily  phone  calls  and  In  Process  Reviews  (IPRs)  approximately  every 
six  weeks,  the  finished  products  were  usually  a  surprise  to  the  TTFA  staff. 
What  they  received  seldom  appeared  to  incorporate  changes  that  they  re¬ 
quested. 

Thirdly,  the  BNCOC  instructional  personnel  only  saw  the  final  products,  a 
situation  which  could  have  only  increased  their  perceptions  that  the  TTFA 
programs  were  imposed  solutions.  Daily  contacts  with  the  instructional  per¬ 
sonnel  by  the  contractors  would  have  eased  these  fears  and  would  have  allowed 
the  contractors  to  have  a  better  understanding  of  some  of  the  problems  faced 
by  the  BNCOC  instructors.  For  example,  these  contractors  could  have  limited 
the  length  of  the  courseware  because  they  would  have  realized  the  scheduling 
constraints  faced  by  the  BNCOC  instructors. 

Having  the  courseware  developers  at  a  remote  sites  also  caused  problems 
with  processing  the  courseware  into  the  MicroTICCIT.  For  one  thing,  course¬ 
ware  and  software  bugs  existed  In  the  TTFA  hardware  system  which  were  not 
present  in  the  hardware  system  used  by  the  courseware  developers.  The  site 
manager  experienced  problems  with  having  the  courseware  developers  explain  by 
telephone  how  to  fix  these  bugs.  An  on-site  representative  who  was  familiar 
with  the  courseware's  program  could  have  quickly  fixed  most  courseware  prob¬ 
lems.  The  on-site  representative,  who  would  have  been  familiar  with  the 
MicroTICCIT  System’s  software,  could  have  helped  the  site  manager  correct 
software  bugs.  Such  software  bugs  are  hard  to  fix  with  telephoned  instruc¬ 
tions  . 

Recommendations :  A  courseware  development  contract  should  specify  that 

the  courseware  developer  be  located  or  have  a  representative  at  the  instruc¬ 
tional  site.  This  person  should  remain  at  this  site  until  the  courseware  is 
fully  implemented. 


Summary 

This  section  has  described  some  of  the  problems  encountered  by  TTFA  in 
managing  CBI  projects.  As  discussed,  the  following  lessons  were  learned  from 
TTFA' s  experiences  regarding  managing  the  CBI  program:  1)  a  single  project 
manager  must  be  employed;  2)  each  participant  must  understand  the  constraints 
faced  by  the  other  participants;  3)  all  key  participants  must  be  located  at 


the  instructional  site;  4)  the  CBI  site  manager  must  be  thoroughly  trained 
before  the  project  starts;  and  5)  the  project  manager  must  choose  a  site 
which  needs  few  alterations. 


DEVELOPMENTAL  ISSUES 


The  development  of  any  Army  training  program  should  follow  the  Systems 
Approach  to  Training  (SAT — TRADOC  Reg  350-7).  This  SAT  process  dictates  that 
the  training  program's  development  follow  a  specific  developmental  sequence. 
TTFA  has  encountered  problems  when  the  SAT  sequence  is  not  followed. 


Task  Selection  and  Training  Design 

Three  different  selection  techniques  were  used  to  identify  19K  BNCOC 
tasks  for  conversion  to  CBI.  Each  of  these  techniques  is  discussed  in  turn. 

Land  Navigation:  Land  Navigation  was  identified  very  early  in  this  pro¬ 
ject  as  a  training  problem  area.  Representatives  of  the  TTFA  decided  that 
the  first  effort  to  convert  traditional  instruction  to  CBI  would  come  in  the 
land  navigation  area  because  of  existing  problems  with  the  traditional  in¬ 
structional  approach.  A  Land  Navigation  Steering  Committee  was  empanelled  to 
supervise  the  work.  This  committee  included  representatives  from  the  Fort 
Knox  NCO  Academy,  the  Directorate  of  Training  and  Doctrine,  Training  Group, 
TRADOC,  ARI ,  and  instructors  from  the  19K  BNCOC  course.  Since  none  of  these 
agencies  had  the  appropriate  resources  at  Fort  Knox  to  author  the  courseware, 
a  contractor  was  let  for  this  purpose.  Task  selection  was  based  on  current 
training  and  performance  deficiencies  and  a  feasibility  analysis  of  using  CBI 
for  training  and  performing  each  task  based  on  procedures  in  Knerr,  Sticha, 
Ramsberger,  Harris  &  Tkacz  (1984). 

Five  tasks  were  selected  for  development: 

Identify  natural  terrain  features  and  determine  elevation. 

Orient  a  map  to  the  ground  by  map-terrain  association. 

Determine  location  on  the  ground  by  terrain  association. 

Locate  an  unknown  point  on  a  map  or  on  the  ground  by  intersection  or 
resection. 

Analyze  terrain  using  the  five  military  aspects  of  terrain. 

Courseware  was  designed  to  solve  terrain  visualization  and  contour  prob¬ 
lems.  A  landform  model  was  developed  for  the  courseware  that  included  vari¬ 
ous  terrain  features.  The  three-dimensional  terrain  model  was  then  marked 
with  contour  patterns.  Video  sequences  of  the  model  were  developed  that 
shifted  from  an  overhead  two-dimensional  view  (map  portrayal)  to  a  horizontal 
perspective  (heads-up  view  of  the  terrain).  A  special  effort  was  taken  to 


provide  students  with  a  dynamic  view  of  real  world  terrain.  Terrain  photo¬ 
graphs  were  taken  that  allowed  students  to  pan  360°  and  to  zoom  in  and  out 
for  close-up  views  (Lickteig  &  Burnside,  1986). 

Training  courseware  for  two  of  the  land  navigation  tasks — Deter¬ 
mine  a  Location  and  Orient  a  Map  by  Terrain  Association — were  not 
formally  evaluated  due  to  inadequate  courseware  design.  The  design 
problem  unique  to  both  of  these  tasks  was  how  to  best  display  a 
360-degree  horizontal  view  of  the  terrain  on  a  single  workstation 
monitor,  a  TV  screen.  The  solution  attempted  in  the  courseware 
design  was  to  simulate  a  visual  pan  across  the  terrain  by  rapidly 
accessing  from  the  videodisc  120  sequentially  stored  video  frames 
with  each  frame  incrementing  the  pan  by  3  degrees.  While  the  pan 
simulation  appeared  realistic,  soldiers  were  unable  to  integrate 
this  linear  set  of  frames  into  a  circular  (360-degree),  composite 
view  of  the  surrounding  terrain.  Soldiers  were  especially  dis¬ 
tressed  by  their  inability  to  infer  the  angular  displacement  between 
terrain  features  that  could  not  be  simultaneously  viewed  on  the 
monitor. 

The  design  problem  unique  to  both  of  these  tasks  was  the  re¬ 
quirement  to  maintain  orientation  of  a  360-degree  horizontal  pan  of 
the  terrain  when  presented  on  a  single  monitor  with  its  limited 
field-o f-view.  The  requirement  was  analogous  to  a  tank  commander's 
task  of  viewing  the  terrain  while  looking  through  an  isolated  vision 
block  as  the  tank's  turret  is  rotated.  This  formidable  requirement 
in  the  tank  environment  was  compounded  in  the  courseware  design  by 
the  absence  of  correlated  vestibular  cues,  since  the  viewer  remained 
stationary  and  the  terrain  rotated.  Each  of  the  courseware's  360- 
degree  pans  of  the  terrain  were  based  on  120  video  frames  that 
incremented  the  soldier's  field  of  view  by  3  degrees  as  he  "rotated" 
to  view  the  surrounding  landscape.  Even  when  soldiers  were  able  to 
identify,  or  associate,  two  or  more  terrain  features  with  those 
depicted  on  the  topographic  map,  they  had  great  difficulty  inferring 
the  angular  displacement  between  these  features. 

In  anticipation  of  this  problem,  courseware  design  included  an 
icon  of  a  moving  arrow  on  the  display  that  continually  updated  the 
angular  distance  the  soldier  had  rotated  from  the  initial  frame.  At 
the  termination  of  each  pan  (from  0  degrees  to  360  degrees)  initi¬ 
ated  by  the  student,  the  arrow  rotated  and  depicted  the  angular 
displacement  between  the  student's  current  view  and  his  initial  view 
of  the  terrain.  This  solution  was  inadequate  (Lickteig  &  Burnside, 

1986). 

Skill  Level  1  Remedial  Training:  A  second  problem  identified  early  on 
was  the  need  to  convert  as  much  of  the  pre-course  diagnostic  test  as  possible 
to  CBI  and  to  provide  remedial  training  on  the  selected  tasks.  Tasks  in  the 
diagnostic  test  had  been  selected  as  representing  skills/knowledge  needed  in 
the  course  but  not  taught  because  they  were  at  Skill  Level  1.  The  POI  for 


19K  BNCOC  did  not  include  time  for  remediation.  Remediation  was  the  respon¬ 
sibility  of  the  student.  Since  several  of  the  diagnostic  tasks  were  primar¬ 
ily  cognitive,  they  seemed  to  be  natural  selections  for  CBI.  Twenty-one 
tasks  were  selected  as  candidates.  From  this  list,  five  were  selected  for 
CBI  treatment  based  on  testability  and  trainability  with  the  CBI  delivery 
system.  These  tasks  were: 

Determine  grid  coordinates  of  a  point  on  a  military  map  using  the  mili¬ 
tary  grid  reference  system. 

Communicate  using  visual  signaling  techniques. 

Recognize  and  identify  friendly  and  threat  armored  vehicles. 

Establish  tank  firing  positions. 

Operate  radio  set. 

Training  design  for  these  tasks  followed  the  Training  Effectiveness  and 
Cost  Effectiveness  Prediction  (TECEP)  model  (Knerr,  Nadler,  &  Dowell,  1984). 
TECEP  translates  task  descriptions  and  learning  principles  into  prescriptions 
for  training.  TECEP  specified,  for  each  type  of  task,  design  considerations, 
including  practice,  feedback  and  reinforcement,  guidance  and  prompts,  learn¬ 
ing  strategies,  and  changes  in  the  training  design  to  enhance  stages  of 
learning  (Knerr  et  al,  1986). 

TECEP  is  one  model  which  could  be  used  to  provide  a  better  basis  for  task 
selection  (and  design)  than  the  much  more  informal  steering  committee  ap¬ 
proach.  All  five  of  these  tasks  translated  well  to  CBI  and  were  subsequently 
shown  to  provide  adequate  student  learning. 

Communlcatlons-Electronics  Operating  Instructions,  Call  For/Adjust  Indi¬ 
rect  Fire,  NBC  Defense,  and  Mine  Warfare:  Normally,  media  are  selected  by 
identifying  the  instructional  requirement  of  a  task  and  then  matching  the 
identified  characteristics  to  the  characteristics  of  the  available  media. 
Factors  such  as  the  instructional  characteristics  of  the  learner,  and  in¬ 
structional  strategies  are  included  (Reiser  &  Gagne,  1983). 

In  this  project,  the  medium  had  already  been  selected  —  a  CBI  delivery 
system  with  videodisc  capability.  The  task  here,  then,  was  to  identify  the 
characteristics  of  the  medium  and  select  the  objectives  that  could  best  be 
presented  using  this  system,  i.e.,  apply  the  media  selection  process  back¬ 
wards  (Jay,  1986). 

The  media  selection  process  was  broken  down  into  three  major  steps. 

Step  I.  Determine  objective  performance  level.  Objectives,  enabling 
objectives,  testing  conditions,  and  criteria  were  used  to  determine  the  per¬ 
formance  level  of  each  objective  as  currently  taught. 

Step  2.  Apply  media  selection  model.  Reiser  and  Gagne's  (1983) 
model  was  selected.  Since  the  model  narrowly  defines  CBI,  some  portions  of 
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it  were  not  applicable.  Prior  to  application  of  the  model,  information  was 
collected  about  each  objective  in  the  course.  Each  objective  was  analyzed 
for  information  required  by  the  selection  model;  performance  level  (crawl, 
walk,  run)  of  each  objective  was  determined,  preliminary  designs  of  CBI  les¬ 
sons  were  sketched  out,  and  instructional  requirements  were  specified.  This 
information  was  put  through  the  media  selection  flowchart.  Finally,  possible 
instructional  design  strategies  were  laid  out. 

Step  3.  Apply  the  final  selection  process.  Final  recommendations 
were  based  on  a  mix  of  objectives  which  could  be  completed  in  the  time  and 
budget  allotted,  priority,  and  student  GO  rates.  Three  factors  were  consid¬ 
ered  when  determining  the  time/cost  to  convert  to  CBI;  performance  level  of 
each  objective,  instructional  design/features,  and  whether  the  instructional 
model  had  to  be  custom-designed  and  coded  or  an  existing  model  could  be 
adapted.  Performance  levels  and  preliminary  instructional  design  sketches 
were  compared  with  the  relative  cost  of  the  required  instructional  features 
and  whether  a  new  instructional  model  would  be  required  (Jay,  1986).  This 
process  resulted  in  the  selection  of  the  following  tasks. 

Communications-Electronics  Operating  Instructions 

Call  For/Adjust  Indirect  Fire 

NBC  Defense 

Mine  Warfare 

Recommendations :  Task  selection  for  CBI  treatment  must  be  done  either  by 
instructional  developers  thoroughly  versed  In  this  process  and  experienced  in 
CBI  development  or  by  the  careful  application  of  a  CBI  task  selection  model. 
Careful  task  selection  can  be  undone  by  poor  CBI  design,  hence  CBI  design 
must  be  developed  by  personnel  with  experience.  Since  CBI  courseware  design 
appears  to  be  a  much  more  exact  process  than  the  design  of  traditional  in¬ 
struction,  considerable  time  and  resources  must  be  allotted.  Problematic 
tasks  that  require  complicated  cognitive  functions  should  be  avoided  unless 
the  literature  can  show  previous  successful  application. 


Selection  of  a  Training  Medium 


Fort  Knox's  TTFA  was  In  an  early  developmental  box  because  of  an  early 
decision  to  use  CBI  as  the  training  medium.  This  medium  seemed  to  be  appro¬ 
priate  to  this  project's  training  needs.  As  previously  indicated,  the  first 
phase  of  the  SAT  sequence — job  and  task  analyses — determined  that  new  in¬ 
structional  programs  were  needed  In  the  19K  BNCOC  program  of  instruction 
(POI)  on:  a)  Call  for  Indirect  Fire;  b)  Land-Navigation;  c)  Military  Commu¬ 
nications;  d)  Mine  Warfare;  e)  NBC  Defense;  f)  Remediation  Training  (for 
BNCOC  students  who  had  problems  with  prerequisite  skills);  and  g)  Tank  Fire 
Commands.  CBI  seemed  to  be  especially  appropriate  for  training  soldiers  in 
these  different  lessons,  CBI  could  provide  standardized  training  which  was 
missing  in  many  BNCOC-level  training  programs.  CBI  could  also  allow  military 
students  to  have  the  needed  practice  when  instructor  resources  were  low. 
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After  completing  the  courseware,  TTFA  discovered  that  some  aspects  of 
these  tasks  were  not  good  candidates  for  CBI  training  (see  the  section  on 
Task  Selection  and  Training  Design).  TTFA  personnel  have  also  observed  that 
procedural  skill  tasks  which  involve  relatively  stable  instructional  proce¬ 
dures  seem  to  be  the  most  suitable  for  CBI  training.  For  example,  training 
students  to  determine  grid  coordinates  is  especially  suited  for  CBI  training. 
The  instructional  procedures  for  training  this  particular  skill  have  not 
markedly  changed  in  a  number  of  years  and  this  task  is  not  likely  to  change 
in  the  future.  Conversely,  TTFA  has  discovered  that  tasks  (e.g.,  tactics) 
subject  to  change  because  of  continually  changing  equipment  or  doctrine  are 
not  good  candidates  for  CBI  training. 

Unfortunately,  a  large  number  of  tasks  selected  for  CBI  conversion 
changed  during  the  time  period  between  selection  and  implementation.  It  was 
never  perfectly  clear  whether  these  tasks  had  been  incorrectly  defined  in  the 
beginning  or  had  actually  changed  during  program  development.  Some  task 
procedures  also  changed  because  a  new  SHE  determined  that  the  old  SMEs  opin¬ 
ions  were  inaccurate.  The  management  team  was  perhaps  naive  to  have  assumed 
that  two  or  more  SMEs  would  agree  in  all  particulars  about  the  performance  of 
a  task.  These  problems  with  task  stability  thus  underscore  the  need  for  a 
stable  team  when  producing  CBI  courseware. 

Selecting  an  inappropriate  task  for  CBI  use  has  more  dire  consequences 
than  selecting  an  inappropriate  task  for  other  delivery  systems.  For  one 
thing,  it  is  more  labor  intensive  and  difficult  to  change  a  CBI  course  than 
to  change  a  platform  instruction  course.  TTFA  has  found  that  a  platform 
lecture  can  be  changed  in  matter  of  minutes  or  days  if  it  proves  ineffective. 
A  similar  change  in  a  CBI  lesson  may  take  weeks  or  months.  For  example,  TTFA 
discovered  that  changing  one  question  on  a  computer-embedded  practical  exer¬ 
cise  involved  several  modifications  In  the  courseware  program.  The  operator 
had  to  change  several  things — the  question,  answer  key,  correct  answer  feed¬ 
back,  and  wrong  answer  feedback — which  were  programmed  as  separate  bits  of 
information.  The  operator  also  had  to  make  sure  that  the  proper  procedures 
for  making  changes  were  followed  otherwise  the  students  might  not  receive  the 
question  or  corresponding  feedback.  Selecting  an  inappropriate  task  for  CBI 
use  also  jeopardizes  future  CBI  funding.  Funding  agencies  would  be  reluctant 
to  invest  the  large  sums  needed  in  CBI  training  projects  without  being  as¬ 
sured  of  positive  results  for  their  investments. 

TTFA,  however,  was  forced  to  add  instructional  objectives  after  the 
courseware  was  nearly  developed.  Such  changes  were  made  to  correspond  to 
changes  made  to  the  BNCOC  training  program.  Since  our  courseware  was  viewed 
as  an  exploratory  project  by  the  Armor  School,  the  new  BNCOC  objectives  did 
not  necessarily  correspond  to  the  developed  courseware.  These  changes  were 
made  to  tasks  with  traditionally  stable  subject  matter.  For  example, 
encrypting  grid  coordinates  was  a  new  objective  for  the  Military  Communica¬ 
tions  tasks  which  was  not  included  in  the  developed  courseware.  Major  modi¬ 
fications  then  had  to  be  made  to  this  courseware. 

Recommendations :  The  decision  to  use  a  particular  training  medium  is 
best  made  in  the  third  SAT  phase — the  developmental  phase.  The  instructional 
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medium  must  be  appropriate  for  training  the  instructional  objectives  which 
were  developed  in  the  second  SAT  phase — the  design  phase.  Finally,  the  data 
from  the  CBI  literature  should  be  considered  in  determining  the  probability 
of  success  in  using  CBI  for  the  selected  objectives.  Correspondingly,  deci¬ 
sions  made  during  the  design  phase  must  remain  stable  throughout  the  develop¬ 
mental  process. 

Selection  of  the  Hardware  System 

Fort  Knox's  TTFA  program  was  also  in  a  developmental  box  because  of  an 
early  selection  of  an  instructional  delivery  system.  The  particular  hardware 
system  selected  seemed  to  be  an  excellent  choice  for  the  19K  BNCOC  training. 
This  hardware  system  had  an  excellent  graphics  capability,  was  videodisc 
based,  allowed  full  student  interaction  through  a  keyboard  or  a  light  pen, 
and  each  host  station  could  operate  up  to  forty  student  terminals.  Also, 
previous  research  found  th*  system  to  be  cost-efficient  for  military  train¬ 
ing  purposes  (Graham,  Shlechter,  &  Goldberg,  1986). 

TTFA  then  found  that  the  capabilities  of  the  delivery  system  drove  the 
instructional  design.  Tasks  were  selected  for  conversion,  and  the  corre¬ 
sponding  instructional  design  was  determined  based  upon  beliefs  about  what 
the  CBI  system  could  deliver.  Because  the  system  appeared  to  be  enormously 
capable,  risky  tasks  were  selected  (e.g.,  the  Land  Navigation  tasks)  and 
problematic  training  procedures  were  designed,  such  as  those  used  in  the  map 
orienting  tasks. 

Another  risky  training  procedure  involved  the  use  of  the  voice  recogni¬ 
tion  system  to  train  Tank  Fire  Commands  and  some  aspects  of  Military  Communi¬ 
cations.  All  the  expert  judgment  that  was  solicited  concerning  device 
capabilities  assured  the  TTFA  team  that  the  hardware  could  handle  it.  Unfor¬ 
tunately,  several  problems  surfaced  with  the  interface  between  the  voice 
recognition  system  and  the  MicroTICCIT  System  II.  For  one  thing,  this  voice 
system  added  peripherals  to  the  MicroTICCIT  that  stretched  the  system  beyond 
its  designed  capabilities.  Also  the  courseware  developer  had  problems  be¬ 
cause  the  length  of  the  string  of  utterances  for  Military  Communications 
lessons  were  much  longer  than  anticipated.  A  misrecognition  anywhere  in  the 
string  resulted  in  a  misrecognition  for  the  entire  string.  Consequently,  the 
Military  Communications  program  was  beset  by  delays  and  other  problems.  It 
was  ultimately  decided  to  complete  the  Military  Communications  program  with¬ 
out  using  the  voice  recognition  system. 

Recommendations :  The  decision  to  use  a  particular  CBI  system  should  be 
made  after  the  instructional  objectives  have  been  determined.  Otherwise, 
hardware  limitations  rather  than  the  instructional  objectives,  determine 
courseware  development.  The  Army  should  be  aware  of  this  problem  when  it 
selects  and  mandates  a  standard  Instructional  system  for  all  users. 


Courseware  Production  Methods 


The  method  of  producing  the  courseware  also  effected  the  instructional 
program.  One  courseware  contractor  was  concerned  with  production  efficiency. 
This  firm  thus  took  steps  to  produce  the  required  courseware  within  the  con¬ 
tractual  time  frame.  This  contractor  designated  a  production  control  monitor 
who  tried  to  make  sure  that  the  courseware  was  developed  on  time  and  within 
budget.  Different  people  worked  on  different  aspects  of  the  same  courseware. 
And  a  "template"  method  was  used  to  produce  the  courseware.  This  method 
involved  using  the  same  instructional  design  for  similar  courseware.  The 
template  method  also  involved  creating  a  standard  page  design  which  could  be 
used  for  any  number  of  CBI  programs.  For  example,  each  TTFA  lesson  contained 
the  same  "help"  page  for  using  the  system.  These  approaches,  especially  the 
template  method,  were  used  to  reduce  the  costs  and  time  involved  in  creating 
the  CBI  courseware. 

These  approaches  caused  a  number  of  problems  for  TTFA.  First  of  all, 

TTFA  personnel  and  the  military  SMEs  were  asked  to  review  the  interim  prod¬ 
ucts — courseware  scripts — as  rapidly  as  possible.  Sometimes  these  scripts 
were  as  long  as  fifty  pages  per  segment.  They  also  received  a  script  for 
each  completed  segment  rather  than  receiving  an  entire  package  of  scripts  for 
a  particular  piece  of  courseware.  As  indicated  in  the  team  approach  section, 
the  SMEs  could  not  review  the  courseware  in  the  contractors'  time  frame.  The 
pressure  placed  upon  them  by  the  contractor  to  review  the  scripts  made  the 
SMEs  more  reluctant  to  thoroughly  review  the  different  scripts.  Also,  these 
SMEs  had  trouble  visualizing  the  CBI  courseware  from  the  "hard-copy"  scripts. 
The  SMEs  continually  complained  that  the  on-line  courseware  did  not  look  like 
the  previously  reviewed  scripts.  TTFA's  project  manager  and  his  staff  for 
developing  the  courseware  also  experienced  similar  problems  with  reviewing 
the  "hard-copy"  scripts.  The  developed  courseware  thus  contained  content 
errors.  For  instance,  erroneous  coding  procedures  were  discovered  on  the 
developed  Military  Communications  courseware. 

The  template  method  also  caused  problems  in  presenting  some  of  the  in¬ 
structional  materials.  The  "help"  pages  explained  the  mechanisms  for  going 
to  different  pages  within  this  courseware  instead  of  providing  any  help  with 
the  instructional  content.  Soldiers  completing  the  Military  Communications 
courseware,  for  example,  were  upset  that  these  help  pages  did  not  provide  any 
additional  information  about  the  procedures  for  encoding  messages.  The  tem¬ 
plate  method  also  limited  the  amount  of  screen  space  available  for  presenting 
the  instructional  information.  Multiple-choice  items,  for  example,  were 
limited  to  a  few  briefly  stated  alternatives.  Changing  the  developed  multi¬ 
ple-choice  items  was  possible  only  after  considerable  time  and  effort  was 
spent  by  the  terminal  site  manager  in  changing  these  pages.  The  template 
method  also  forced  a  non-optimal  instructional  design  upon  some  courseware. 
For  example,  the  same  instructional  design  was  used  for  two  similar  appearing 
tasks — using  the  Radiacmeter  and  Dosimeter — even  though  these  two  BNCOC 
courses  contained  different  subject  matter  and  testing  methods. 

Time  delays  caused  by  the  need  to  mail  scripts  back  and  forth  produced  a 
need  to  rapidly  produce  the  courseware.  Computer  screens  were  designed  with 
too  much  haste  as  was  the  editing  of  textual  materials  and  graphics.  It  was 


also  obvious  that  different  people  produced  different  aspects  of  the  same 
program.  Different  styles  of  presenting  materials  were  found  within  a  par¬ 
ticular  courseware.  For  example,  different  feedback  procedures  for  similar 
types  of  items  were  found  throughout  the  Call  for  Fire  program. 

Despite  everyone’s  efforts,  the  development  of  the  CBI  programs  did  not 
proceed  on  a  time  scale  parallel  to  previous  training  developments.  CBI 
development  took  much  longer.  The  TTFA  personnel  were  unprepared  for  this. 
Pressures  for  more  rapid  courseware  mounted  from  both  the  contractor  and  the 
funding  agency.  The  contractor,  of  course,  was  worried  about  arriving  at  the 
end  of  the  contract  without  having  satisfied  all  contractual  requirements. 

The  funding  agency  was  concerned  with  a  steady  outpouring  of  funds  without 
any  implemented  programs  to  show  for  it.  As  previously  indicated,  all  the 
CBI  programs  suffered  because  of  these  pressures. 

Recommendation :  Training  effectiveness  must  be  the  bottom  line  for 
courseware  production.  SMEs  and  the  courseware  reviewers  must  be  given  in¬ 
terim  products,  including  scripts,  learning  maps,  and  test  plans,  which  are 
easy  to  follow.  These  people  must  be  allowed  an  adequate  amount  of  time  to 
review  these  products.  In  fact,  all  parties  to  the  CBI  development  and  im¬ 
plementation  process  must  appreciate  that  fielding  these  programs  is  a  slower 
process  than  traditional  training  program. 

We  also  recommend  that  the  courseware  developer  employ  a  quality  control 
person.  This  individual  must  check  and  re-check  each  on-line  item  for  con¬ 
tent  and  typographical  errors  and  must  make  sure  that  a  consistent  style  Is 
maintained  for  each  course.  This  individual  must  be  independent  of  the  de¬ 
signing  and  programming  process  because  individuals  responsible  for  develop¬ 
ing  any  textual  product  have  trouble  with  proofreading  the  materials.  And 
the  courseware  monitoring  team  and  the  SMEs  must  review  the  on-line  course¬ 
ware  for  content  and  design  errors  before  It  Is  released.  TTFA  found  that 
controlling  such  content  errors,  including  typographical  errors,  was  very  im¬ 
portant  because  military  instructors  were  hesitant  to  implement  courseware 
with  even  the  slightest  content  errors. 


Summary 

Quality  CBI  development  thus  requires  that  the  utmost  care  be  given  to 
each  developmental  sequence.  Choosing  a  training  medium,  a  particular  sys¬ 
tem,  and  a  particular  instructional  design  should  be  based  upon  instructional 
objectives.  Quality  control  is  also  required  for  producing  a  quality  course¬ 
ware. 


IMPLEMENTATION  ISSUES 

A  successful  CBI  implementation  demands  that  the  training  program  become 
a  standard  piece  of  the  program  of  instruction  (POI)  or  curriculum.  A  CBI 
program,  regardless  of  its  training  effectiveness,  will  not  reach  this  in¬ 
structional  status  if  the  program  does  not  satisfy  the  users. 
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Needs  of  the  Instructors 


The  TTFA  project  experienced  Implementation  problems  because  the  BNCOC 
instructors  were  reluctant  to  implement  courseware  which  did  not  fully  meet 
their  immediate  needs.  For  example,  the  BNCOC  instructors  felt  that  some 
portions  of  the  Call  for  Fire  program  would  be  rarely  used  as  some  of  these 
materials  were  too  advanced  for  BNCOC  students.  On  the  other  hand,  TTFA  has 
found  that  instructions  are  eager  to  implement  any  CBI  programs  which  will 
meet  an  instructional  need.  For  example,  an  instructor  from  a  more  advanced 
Armor  School  course  was  eager  to  use  the  Call  for  Fire  program  as  his  stu¬ 
dents  needed  all  the  practice  possible  on  this  task. 

TTFA  also  failed  to  calm  the  instructors'  fears  about  being  replaced  by 
the  CBI  system.  At  an  Armor  School  meeting,  for  example,  the  representative 
for  the  BNCOC  instructors  complained  that  CBI  training  was  not  needed  as  his 
instructors  were  better  prepared  to  teach  the  different  materials.  His  com¬ 
ments  seemed  to  be  based  on  the  need  to  save  his  instructors'  jobs.  The 
BNCOC  instructors'  fears  were  thus  a  barrier  to  implementing  the  TTFA  pro¬ 
gram. 

Recommendation :  Instructional  personnel  must  have  an  active  role  in  CBI 

development.  They  must  assume  "ownership”  of  the  programs  developed. 


Implementation  and  Instructional  Management  Problems 


Instructional  management  was  a  continual  problem.  For  one  thing,  in¬ 
structors  did  not  want  any  program  which  they  felt  was  too  difficult  for 
their  students  to  complete.  A  concern  was  expressed  that  these  programs 
contained  too  much  reading  material  for  their  students.  Another  concern  was 
that  difficult  courseware  would  affect  the  students'  desire  to  complete  the 
program.  The  Military  Communications  courseware,  for  example,  contained  some 
items  which  frustrated  the  students  because  they  could  not  be  answered.  TTFA 
personnel  noticed  that  the  classroom  scene  became  quite  tense  when  the  stu¬ 
dents  attempted  to  complete  these  items.  Some  students  even  exhibited  some 
verbal  hostility  toward  the  program.  Fortunately,  the  classroom  proctor  was 
able  to  help  the  students  complete  these  items  without  too  much  time  being 
wasted.  The  proctor  was  able  to  provide  this  help  because  of  his  familiarity 
with  this  particular  courseware. 

Monitoring  the  students'  time  at  a  CBI  terminal  was  another  instructional 
management  problem.  TTFA  did  not  initially  realize  the  students'  limited 
capacity  for  continually  viewing  a  CBI  program.  Students  were  initially 
given  breaks  as  given  in  traditional  platform  instruction  courses — after  each 
fifty  minutes  of  instruction.  TTFA  personnel  found  that  the  evaluation  sub¬ 
jects  and  BNCOC  students  would  start  yawning  and  losing  their  concentration 
after  thirty  minutes  of  continuous  CBI  training.  To  maintain  the  instruc¬ 
tional  flow,  instructors  should  allow  their  students  to  have  breaks  more 
often  than  the  breaks  provided  in  traditional  platform  learning. 
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Scheduling  was  also  effected  by  the  number  of  terminals  available  for 
student  use.  For  example,  the  current  19K  BNCOC  class  offered  at  Fort  Knox 
ranges  in  size  from  eight  to  sixteen  students  per  cycle.  Because  of  con¬ 
straints  beyond  TTFA's  control,  this  project  had  only  four  student  terminals. 
The  BNCOC  instructors,  operating  under  tight  scheduling  constraints,  were  not 

able  to  schedule  all  students  to  receive  CBI  training.  Such  limited  terminal 

availability  also  presents  problems  for  instructors  without  any  scheduling 
constraints.  These  trainers  must  find  activities  for  students  who  are  not 
using  the  terminals  while  other  students  are  engaged  in  CBI  training. 

If  an  inadequate  number  of  terminals  are  available,  then  small  group  (two 

or  more  at  a  terminal)  training  is  possible  option.  Shlechter  (1987a)  found 
that  students  who  received  group  CBI  training  completed  the  lesson  quicker 
than  did  the  student  who  received  individual  CBI  training.  He  did  not  find 
any  differences  in  the  subjects'  learning  of  the  materials.  Group  training 
would  thus  allow  more  students  to  receive  CBI  training  with  the  same  among  of 
hardware.  Of  course,  group  presentation  of  CBI  materials  may  not  be  appro¬ 
priate  for  certain  types  of  courseware,  e.g.,  a  remediation  program. 

Recommendations :  The  implementation  team  must  devise  a  plan  for  dealing 
with  potential  course  management  problems.  Instructors  need  to  know  that 
more  frequent  breaks  should  be  scheduled.  Enough  terminals  are  needed  to 
obviate  scheduling  problems.  If  it  is  not  possible  to  acquire  an  adequate 
number  of  terminals  then  group  presentation  of  CBI  materials  is  recommended. 


Implementation  Plans  and  Maintaining  the  System 

TTFA  has  also  discovered  that  any  system  failure  can  greatly  disrupt  the 
instructional  process.  For  example,  a  faulty  light-pen,  the  MicroTICCIT 
system's  responding  mechanism,  impeded  students'  ability  to  complete  a  les¬ 
son.  Such  minor  system  problems  were  not  uncommon.  The  system's  reliability 
also  effected  the  developmental  and  implementation  process.  Our  videodisc- 
based  system — as  do  all  systems — had  some  hardware  and  software  bugs.  For 
example,  TTFA  personnel  experienced  problems  with  frozen  computer  frames, 
(i.e.,  the  terminal  getting  stuck  on  one  page).  Developmental  and  implemen¬ 
tation  delays  were  then  caused  when  the  system  was  down  for  an  extended  pe¬ 
riod  of  time. 

Such  reliability  problems  have  also  been  shown  by  the  research  literature 
to  be  an  important  factor  in  instructors'  decisions  to  continue  or  discon¬ 
tinue  using  the  system  (Shlechter,  in  press).  For  one  thing,  a  relationship 
has  been  shown  to  exist  between  educational  productivity  and  a  medium's  down¬ 
time  during  the  instructional  process  (Shlechter,  1987).  Also,  instructors 
must  re-schedule,  if  possible,  their  program  to  meet  the  changes  necessitated 
by  the  problems  with  the  system. 

Recommendations ;  Implementation  plans  should  include  procedures  for 
maintaining  the  system.  A  maintenance  contract  with  the  hardware  developer 
is  an  absolute  necessity.  The  site  manager  is  also  important  for  maintaining 
the  system.  Many  systems  problems  are  resolvable  through  "quick-fix"  proce¬ 
dures  (e.g.,  activating  certain  programming  codes  to  unlock  a  frozen  frame). 


A  successful  implementation  requires  thoroughly  training  the  system  operator 
in  troubleshooting  the  CBI  system. 


A  successful  implementation  also  requires  choosing  a  CBI  hardware  con¬ 
figuration,  including  its  peripherals,  with  a  proven  track  record.  TTFA's 
system  was  problematic  because  many  new  features  were  added  to  the 
MicroTICCIT  System  II. 


Training  the  Instructors 


TTFA's  implementation  plans  should  have  also  included  an  instructional 
program  for  the  BNCOC  instructors.  These  instructors  were  given  some  pre¬ 
liminary  training  on  the  courseware  and  MicroTICCIT  system  by  the  implementa¬ 
tion  team.  However,  this  training  was  not  sufficient  for  them  to  resolve 
many  problems  without  a  civilian  proctor's  being  present. 

A  trained  instructor  would  be  able  to  resolve  quickly  any  minor  system 
problems  before  any  serious  instructional  problems  (e.g.,  bored  students 
bothering  other  students)  could  occur.  Secondly,  the  instructor  needs  to  be 
familiar  with  the  instructional  program  to  maintain  the  instructional  flow. 

As  stated,  the  trained  TTFA  proctor  was  able  to  help  students  complete  the 
Military  Communications  lesson.  The  BNCOC  instructors  should  then  have  been 
"computer  literate"  with  regards  to  the  different  courseware  and  the  Micro¬ 
TICCIT  System  II. 

This  training  for  instructors  should  also  prepare  them  for  a  new  instruc¬ 
tional  role.  Instructors,  especially  military  instructors,  are  quite  famil¬ 
iar  with  the  lecture  mode  associated  with  conventional  platform  training.  As 
discussed,  however,  the  instructor's  role  in  CBI  training  changes  to  that  of 
an  instructional  facilitator  with  the  main  responsibility  of  ensuring  the 
courseware's  proper  instructional  flow.  Instructors  with  such  computer 
literacy  would  thus  not  harbor  as  many  anxieties  about  using  this  training 
medium  because  they  would  know  that  they  still  had  a  viable  instructional 
role.  Instructors  who  are  expecting  too  much  from  CBI  may  become  disillusion¬ 
ed  after  using  this  training  medium. 

Recommendations :  A  successful  implementation  requires  training  the  in¬ 
structors  to  use  the  computer  courseware  and  system  and  adopt  a  new  instruc¬ 
tional  role.  However,  the  personnel — implementation  team  members  and  the 
terminal  site  manager — who  are  responsible  for  this  training  should  refrain 
from  overselling  the  product  to  the  instructional  personnel 


Commitment  from  the  School's  Management 


Many  of  the  Armor  School's  management  personnel  were  not  actively  in¬ 
volved  with  this  project.  These  department  heads  had  been  told  that  TTFA  was 
an  experimental  project  whose  products  would  probably  never  be  permanently 
implemented  at  Fort  Knox.  These  department  heads,  including  the  BNCOC  Divi¬ 
sion  Chief — were  thus  not  totally  committed  to  implementing  this  project's 
products . 


Another  related  implementation  problem  was  that  the  school's  management 
redesigned  the  19K  BNCOC  course  midway  through  the  UFA  project.  The  TTFA 
staff  was  not  notified  of  this  action.  This  problem  underscores  the  continu¬ 
ing  communication  problem. 

This  lack  of  committment  from  the  school's  departmental  heads  was  mani¬ 
fested  in  several  other  ways.  Several  departments  were  hesitant  to  approve 
aspects  of  the  courseware  as  part  of  the  regular  BNCOC  program  of  instruc¬ 
tion.  For  example,  the  Department  of  Evaluation  and  Standardization  (DOES) 
did  not  see  a  need  to  expedite  their  approval  of  the  computer-embedded  tests. 
At  a  military  training  site,  tests  used  to  evaluate  students  must  be  approved 
by  DOES.  The  BNCOC  instructors  could  not  use  these  computer-embedded  tests 
as  evaluation  instruments.  Also,  the  military  SMEs  viewed  their  job  with 
TTFA  as  tangential  to  their  other  military  duties.  Even  though  these  person¬ 
nel  were  extremely  cooperative,  their  TTFA  duties  were  always  secondary  to 
other  duties — e.g.,  creating  their  own  courses.  Implementation  delays  thus 
resulted  in  waiting  for  these  SMEs'  reviews  of  the  on-line  courseware. 

Recommendations ;  The  TTFA  personnel  should  have  spent  more  time  at  the 
project's  beginning  with  these  department  heads  without  overselling  the  new 
training  products.  The  department  heads  may  have  then  been  convinced  of 
TTFA's  viability.  Consequently,  the  management  personnel  would  have  made 
sure  that  the  TTFA  training  programs  were  put  on  their  project  list.  This 
step  is  important  because  such  projects  get  a  permanent  action  officer.  This 
action  officer  would  want  to  develop  a  quality  product  in  a  timely  fashion. 
The  action  officer  would  make  sure  that  his  or  her  personnel  were  committed 
to  implementing  TTFA’s  products.  He  or  she  would  make  sure  that  the  depart¬ 
mental  heads  were  regularly  briefed  about  the  project's  progress. 


Summary 


TTFA  has  found  that  an  effective  CBI  implementation  program  must  include 
procedures  for  (1)  meeting  the  needs  of  the  instructors;  (2)  training  the 
instructors  to  use  the  medium;  (3)  choosing  a  proven  hardware  configuration; 
and  (4)  maintaining  the  instructional  flow.  The  classroom  instructor  and 
on-site  support  staff  also  have  important  roles  in  successfully  implementing 
this  medium.  Finally,  implementation  planning  should  have  the  full  "bless¬ 
ings"  of  the  school's  management  and  instructional  personnel. 


EVALUATION  OF  THE  CBI  PROGRAM 

CBI's  potential  as  a  training  medium  is  also  dependent  upon  quality  eval¬ 
uation  checks.  These  evaluation  checks  must  be  done  at  each  stage  of  the 
program's  life  cycle. 
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TTFA's  "One-on-One"  Testing  Procedure 


TTFA's  preliminary  evaluation  of  the  developed  courseware  involved  a 
"one-on-one”  testing  procedure  with  one  subject  being  closely  observed  by  an 
evaluator.  The  subjects  were  the  course  instructors  and  six  BNCOC-like  sol¬ 
diers  who  came  from  an  operational  armor  unit.  The  evaluation  team’s  previ¬ 
ous  experience  with  soldiers  from  these  units  suggested  that  their 
performance  would  be  similar  to  the  performance  of  BNCOC  students.  Problems 
encountered  by  a  subject  were  recorded  by  the  observer.  The  subject  and 
observer  also  discussed  each  problem  as  it  was  encountered.  Each  subject  was 
encouraged  to  "think  aloud”  when  completing  this  courseware.  In  the  "think¬ 
ing  aloud”  methodology  subjects  provide  a  running  verbal  account  of  their 
ability  to  complete  the  courseware  (for  a  further  description  of  this  method¬ 
ology  see  Ericsson  and  Simon,  1984).  The  "one-on-one"  testing  approach  pro¬ 
vided  TTFA  with  insights  into  subjects'  performance  which  were  hard  to  obtain 
through  traditional  evaluation  procedures.  For  example,  TTFA  realized  after 
examining  data  for  the  Land  Navigation  courseware  that  students  would  have 
trouble  completing  the  map  orienting  task.  Data  also  provided  insights  into 
the  reasons  for  the  subjects'  problems  with  this  task. 

Recommendations ;  Thorough  content  reviews,  including  "one-on-one"  test¬ 
ing,  should  be  conducted  on  the  interim  products.  The  results  of  these  tests 
should  be  known  to  the  courseware  developer  so  that  they  can  correct  any 
problem  areas  before  delivery  for  in-course  evaluation. 


Developmental  Trials 


TTFA's  subsequent  and  more  formal  evaluation  of  the  CBI  courseware  will 
be  referred  to  as  the  developmental  trials.  Determining  the  quality  of  any 
instructional  program  is  difficult.  TTFA  employed  developmental  trials  to 
determine  the  quality  of  their  CBI  programs.  These  developmental  trials 
involved  a  pretest-posttest  design  to  measure  changes  In  subjects'  perform¬ 
ance  on  representative  achievement  measure(s).  Data  were  also  obtained  on 
the  subjects'  time  for  completing  the  lessons  and  their  attitudes  toward  the 
program.  The  subjects  for  these  studies  were  the  BNCOC-like  soldiers  from 
the  operational  armor  unit. 

The  developmental  trials  were  conducted  under  experimental  conditions, 
the  evaluation  team  controlled  the  instructional  environment.  Members  of  the 
evaluation  team  observed  and  monitored  the  subjects'  behavior  while  they 
completed  the  CBI  lessons  and  tests.  For  example,  a  member  of  this  team  also 
served  as  a  proctor  who  assisted  the  students  with  their  problems.  This 
proctor  also  administered  any  off-line  instruments  (e.g.,  demographic  and 
attitudinal  questionnaires)  to  the  subjects.  These  developmental  trials  did 
indicate  that  the  developed  programs  were  suitable  for  inclusion  in  the  BNCOC 
POI. 


There  were  several  potential  problems  with  these  developmental  trials.  A 
problem  with  these  developmental  trials  was  that  the  experimental  scene  may 
have  produced  artificial  results.  Students  have  been  known  to  perform  dif¬ 
ferently  under  experimental  training  conditions  than  under  regular  training 
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conditions.  Secondly,  these  students'  favorable  comments  about  the  course¬ 
ware  could  have  been  geared  to  please  the  evaluation  team  (see  Shlechter, 

1986  for  a  further  discussion  of  this  latter  possibility).  A  third  problem 
was  that  the  BNCOC-like  subjects'  were  not  comparable  to  the  BNCOC  students. 
Shlechter  (1987b)  noted  that  the  BNCOC-like  students  differed  with  regards  to 
demographic  composition  (such  as  they  had  less  previous  experience  with  using 
the  military  communications  materials)  from  normal  BNCOC  students.  This  prob¬ 
lem  could  have  also  been  an  issue  in  the  "one-on-one"  testing.  Finally, 
doubts  about  the  different  programs'  value  remain  because  the  developmental 
trials  data  were  not  directly  compared  with  baseline  measures  of  a  system's 
effectiveness — such  as  the  subjects'  learning  for  the  already  established 
instructional  program.  Even  though  our  data  compared  favorably  to  those 
provided  for  the  established  program  from  previous  classes,  a  baseline  meas¬ 
ure  was  still  needed  to  determine  the  relative  effectiveness  of  the  CBI 
courseware . 

Recommendations :  We  recommend  that  the  developed  courseware  be  evaluated 
by  an  agency  independent  of  the  development  and  implementation  process. 
Courseware  should  be  evaluated  under  actual  learning  conditions  and  with  the 
students  who  are  going  to  be  using  the  courseware.  Baseline  measures  of  a 
system's  effectiveness  should  also  be  obtained. 

Validation  Check 

TTFA  finally  conducted  a  validation  check  on  the  developed  courseware 
before  implementing  it.  Data  were  collected  when  the  BNCOC  students  were 
using  the  program  as  part  of  their  regular  training.  The  BNCOC  instructors 
controlled  this  instructional  environment  with  only  minimal  concessions  to 
data  collection,  i.e.,  allowing  only  one  observer  to  watch  the  entire  class. 
For  the  most  part,  these  data  were  similar  to  the  developmental  trial  re¬ 
sults.  For  example,  the  data  for  the  Military  Communications'  validation 
check  paralleled  those  obtained  in  the  developmental  trials.  However,  the 
validation  checks  data  did  show  that  some  courseware  and  implementation  modi¬ 
fications  were  still  needed.  For  example,  the  instructors  were  still  depend¬ 
ent  on  TTFA  personnel  to  fix  minor  system  problems. 

The  validation  trials'  (and  developmental  trials')  data  were  reported 
back  to  TTFA  management.  The  question  was  whether  to  proceed  directly  with 
the  implementation,  make  the  appropriate  changes,  or  discontinue  the  program. 
With  the  sizeable  investment  in  developing  the  CBI  program,  the  third  option 
was  unlikely.  Of  course,  negative  evaluation  data  about  a  system's  instruc¬ 
tional  effectiveness  would  have  made  management  reconsider  any  future  CBI 
program  development. 

Also,  very  little  is  known  about  CBI's  worth  for  advancing  the  military 
training  process.  This  is  because  implemented  CBI  training  programs  have 
rarely  been  compared  to  other  implemented  instructional  programs  for  that 
courseware.  Training  developers  and  Instructional  personnel  thus  do  not  know 
if  the  Implemented  CBI  program  is  better  than  other  training  programs  for 
their  intended  purposes.  The  evaluation  team  should  compare  the  implemented 
CBI  program  with  the  already  established  program. 
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Recommendations ;  Questions  will  remain  about  a  program's  effectiveness 
until  the  implemented  program  either  fails  or  becomes  obsolete.  The  evalua¬ 
tion  team  should  collect  data  for  an  extended  period  on  a  system  fully  imple¬ 
mented.  If  possible,  data  should  be  collected  to  compare  CBI  to  any  other 
operational  training  methods  to  determine  the  cost-effectiveness  of  this 
solution.  Such  data  will  show  the  system's  long-term  success  or  failure  for 
training  BNCOC  students.  Data  from  a  fully  implemented  system  will  also 
provide  information  about  a  hardware  system's  reliability  for  normal  training 
conditions,  about  the  relationship  between  projected  and  actual  life  cycle 
costs,  and  about  instructors'  and  students'  actual  attitudes  toward  using  the 
system.  Such  data  are  needed  as  very  little  is  know  about  the  long-term 
worth  of  CBI  systems  in  military  training  programs.  Since  the  primary  func¬ 
tion  of  any  training  program  is  to  insure  that  students  transfer  the  informa¬ 
tion  to  the  appropriate  situations  and  retain  the  information,  transfer 
studies  are  also  needed. 


Summary 


Evaluating  a  CBI  program  has  thus  been  shown  to  be  a  vital  part  of  a 
program's  development  and  implementation  process.  "One-on-one"  testing, 
developmental  trials,  and  validation  checks  are  also  recommended  as  proce¬ 
dures  for  evaluating  CBI  courseware.  Suggested  improvements  in  TTFA's  method 
of  employing  these  evaluation  procedures  are  also  made. 


SUMMARY  OF  RECOMMENDATIONS 


TTFA  did  have  some  success  in  implementing  these  programs.  However,  the 
developmental  program  could  have  been  smoother  if  it  had  been  able  to  antici¬ 
pate  the  problems  discussed  in  this  paper.  In  summary,  the  following  recom¬ 
mendations  have  been  made  based  upon  TTFA's  experiences: 


Managing  the  CBI  Project 


1.  All  agencies  involved  in  CBI  development,  directly  or  tangentially, 
must  meet  regularly  to  share  concerns  and  input  to  the  process. 


2.  All  constraints  to  development  and  implementation  must  be  surfaced  as 
early  as  possible. 


3.  The  time  table  for  development  must  be  more  flexible  than  for  tradi¬ 
tional  instruction. 


4.  Managers  of  the  CBI  development  process  must  be  knowledgeable  of  CBI 
systems  and  courseware  production  methods. 


5.  A  single  project  manager  will  expedite  development  and  evaluation. 
This  person  must  be  the  agent  responsible  for  implementation. 
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6.  The  CBI  site  manager  must  have  previous  CBI  experience.  He/she 
should  be  thoroughly  trained  on  the  CBI  delivery  system  before  system  acqui¬ 
sition. 

7.  The  CBI  site  should  require  few  alterations  to  accept  the  hardware 
and  should  be  located  close  to  or  in  the  "school-house". 

8.  The  CBI  development  manager,  courseware  designers,  courseware  devel¬ 
opers,  instructors,  and  SMEs  should  be  co-located  during  development. 

Developmental 

1.  Task  selection  must  be  done  either  by  instructional  developers  thor¬ 
oughly  versed  in  this  process  or  by  careful  application  of  a  CBI  task  selec¬ 
tion  model. 

2.  Problematic  tasks  that  require  complicated  cognitive  functions  should 
be  avoided  unless  the  literature  can  show  previous  success. 

3.  The  selection  of  training  media  must  follow  the  guidance  in  the  sys¬ 
tems  approach  to  training. 

4.  Data  from  the  CBI  literature  should  be  considered  in  determining  the 
probability  of  success  in  using  CBI  for  selected  objectives. 

5.  Decisions  made  during  the  design  phase  must  remain  stable  throughout 
the  development  process. 

6.  The  decisions  to  use  a  particular  CBI  system  should  be  made  after  the 
instructional  objectives  have  been  determined. 

7.  Training  effectiveness  must  be  the  bottom  line  for  courseware  produc¬ 
tion. 

8.  A  separate,  independent,  person/agency  must  be  employed  for  quality 
control . 

Implementation 

1.  Instructional  personnel  must  have  an  active  role  in  CBI  development. 
They  must  assume  "ownership"  of  the  programs  developed. 

2.  Managers  must  be  prepared  for  instructional  management  problems, 
unique  to  CBI. 

3.  Implementation  plans  must  include  procedures  for  maintaining  hardware 
and  software. 

4.  Instructors  must  be  trained  to  use  the  hardware  early  in  the  develop¬ 
ment  phase. 
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5.  An  action  officer  from  the  school's  Directorate  of  Training  and  Doc¬ 
trine  is  essential  for  smooth  coordination  of  courseware  development. 

6.  Choosing  a  CBI  system  including  its  peripherals  with  a  proven  track 
record  is  recommended. 

Evaluation 

1.  Thorough  content  reviews  should  be  performed  on  interim  products. 

2.  Courseware  evaluation  should  be  performed  by  an  independent  agency. 

3.  Both  formative  and  summative  evaluations  should  be  performed.  Data 
must  be  collected  on  a  fully  implemented  system. 

Conclusion 

This  report  has  also  indicated  that  the  CBI  development  and  implementa¬ 
tion  team  include  personnel  from  all  the  directorates  and  departments  in  the 
school.  This  should  be  done  even  if  these  departments  and  directorates  ap¬ 
pear  to  be  marginally  involved  in  the  effort.  Problems  with  acquiring  a 
terminal  site  and  with  choosing  stable  instructional  materials  were  caused  by 
neglecting  some  of  the  Armor  School's  management  personnel.  Since  CBI  devel¬ 
opment  is  a  very  costly  venture,  it  is  not  worth  risking  unuseable  courseware 
because  some  agencies  have  been  excluded  from  a  meeting  now  and  then.  It  is 
also  imperative  that  agency  representatives  to  the  steering  committee  be 
empowered  to  speak  for  the  agency's  head. 

TTFA  has  thus  found  the  development  and  implementation  of  a  CBI  program 
to  be  more  complex  and  Intricate  than  other  previously  developed  training 
programs.  Quality  CBI  programs  are  still  worth  the  efforts  and  problems  de¬ 
scribed  in  this  report.  This  medium  can  be  useful  to  the  military  as  an 
instructional  tool  and  can  make  the  already  established  program  more  effec¬ 
tive.  This  medium  can  also  alleviate  some  problems  faced  by  the  military 
training  community,  e.g.,  a  need  for  training  basic  skills  with  dwindling 
instructional  resources.  Finally,  CBI  can  provide  the  military  with 
standardization  of  instruction  and  provide  the  reserve  components  with  needed 
sustainment  training. 
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